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VIRTUAL DEVICE ARCHITECTURE and managing data. The storage server according to the 

HAVING MEMORY FOR STORING LISTS OF present invention includes a plurality of communication 

DRIVER MODULES interfaces. A first set of communication interfaces in the 

plurality is adapted for connection to all kinds of users of 

RELATED APPLICATION REFERENCE 5 data. A second set of communication interfaces in the 

plurality is adapted for connection to respective devices in a 
The present application is a continuation-in-part of pnor, storage devices. Data processing resources in the 
co-pending U.S. patent application Ser. No. 09/276,428, storage server arc coupled to the plurality of communication 
entitled Storage Server System, invented by Shari J. Nolan, interfaces for transferring data among the interfaces. The 
Jeffrey S. Nespor, George W. Harris Jr., Jerry Parker Lane, data processing resources comprise a plurality of driver 
and Alan R. Merrcll, filed Mar. 25, 1999 now U.S. Pat. No. ^ modules and configurable logic linking driver modules into 
6,446,141, and such application is incorporated by reference data paths, which are implemented in pairs for redundancy 
as if fully set forth herein. in a preferred system. Each configured data path acts as a 
The present application is related to co-pending U.S. virtual circuit that includes a set of driver modules selected 
patent application Ser. No. 09/346,592, entitled Storage from the plurality of driver modules. A data storage trans- 
Server With Interface to Emulate Legacy Storage Device, action which is received at a communication interface is 
invented by Michael Panas, filed on Jul. 2, 1 999, and owned mapped to one of the configured data paths, 
by the same assignee now and at the time of invention. According to another aspect of the invention, the plurality 

of driver modules includes a protocol server for a protocol 

BACKGROUND OF THE INVENTION supported on a communication interface in the plurality of 

1, Field of the Invention communication interfaces. The protocol server recognizes 

™ . . ^. 1 . i ,u r ij P t * target identifiers which identify particular storage extents 

This invention relates to the field of mass storage systems. ^ , , . ..j- 

, 1 • - 1 * ~t ~*k according to the protocol on the interlace. Iransactions 

In particular, the invcntion-relatcs-to-the-managcment-or-^ , , j . 1 * * i ^ . 

, ^ ' . -.-.i: — 7. r---*-ir;^^t^ addressed to the particular storage extent are mapped to a 

storage transactions m, and-the-configuration-ofrtntelhgent^ . , ^ . , . .u • iu 

** ^ _L r,^ particular configured data path m the server, 

storage arearnetworits. ^ . Z ^ j ■ .i.- . 1 

~~r"tj . , ^. n , . J . ^ The data paths configured in this manner act as virtual 

2. Descnption of the Related Art ^^^^^^^ ^^^.^^^ Users of the data communicate with a 
The storage of large amounts of data in so-caUed mass communication interface on the storage server according to 

storage systems is becoming a common p ractice. Mass a protocol for a particular storage device. Inside the server, 

storage;systems,typically:inchide:storage:devices coupled to^ transactions according to that protocol arc mapped to a 

file:5eryerB:on:data;networks. Users:in-.the:network.commu- virtual storage device implemented by sets of drivers. Set- 

>ni cate-w ith:the:file-seryers:foria cce5S:t o:tfae:data.<The-file^ tjng up and changing the storage tasks performed in a 

scrvcrs-arcitypicaUyrconnectcd-to-specific-5t6rage:dcviccs^ particular data path, and setting up and changing the map- 

y ia-data-c hannelsr>ThezdatazchannelszareiusualIy-imple= p^^g gf ^ storage extent from one data path to another are 

r^mented-with^ point•:to-point3 communicationrP^otocQls accomplished by configuring the sets of driver modules 

designed-for-managmg-storage transactioiis? within the storage server. 

As the amount of storage increases, and the number of file According to an aspect of the invention, the plurality of 

servers in communication networks grows, the concept of a driver modules includes one or more hardware driver mod- 

storag e-area-network^ (SAN)=has arisen. Storagezarea^nct-- ujcs that manage respective communication interfaces, and 

works^^connectza-num^^ one or more internal driver modules that perform data path 

commuriicatioh:;network-which-iscoptimi2edr:forzstorage. i^^ks independently of the plurality of communication inter- 

rtransactions"Por example, fiber channel arbitrated loop faces. The data path tasks include, for example, cache 

FC-AL netw orks are being im plemented as SANs. The memory management, memory mirroring management, 

SANrsupport. many communication sessi6ns^b'etween:users» memory partition management, data migration management, 

of4he:storape-systems:and:the sp ecific 5tora ge:systems:00:> and other tasks for managing storage transactions. By pro- 

the.SANri viding data path tasks of this type in a virtual device 

Filc;servers:an d other i is crs of-th e stora fie-systemsrare architecture, the configuration of the storage system to 

cohfiguHHlo comiiniinicaie-with^pec manage such tasks is essentially transparent to the users. In 

^-J/O thc-storagelsystcnislexpandz addition, providing the virtual device capability at a storage 

✓C^A) ,systcm,:rciconfiguration;is:rcquired:ai:ibe-fil6'^rvers:and. 50 server optimized to perform such tasks allows for improved 

(^^otherusersr^Alsoillif.a need:ari^^^ performance and greater fiexibihty. 

nCK^' de vicc-to anothcrrin:a:so_-callcd data:miEra tion o pera tionrit Also, according to an aspect of the invention, the plurality 

J<^*^ U5;js:ofteuThecess5^ of driver modules includes logic for communicating data 

qg^^AMni^ration^pfoccssTzrAftcr— mi& ration-i within the server environment according to an internal 

r&^nfiguratioQ-atzthe luserz system:: must-be execu ted to. 55 message format. Incoming storage transactions arc trans- 

^ r mak e - theidataiav ailable :f rom:tbc:ne w^dcyicera lated into the internal message format, and placed in the 

Overall, as the complexity and size of storage systems and configured data path for the particular transaction. In one 

networks increase, the problems of managing configuration preferred embodiment, the protocol server performs the 

of the users of the data and of the storage systems them- protocol translation and virtual circuit mapping function, 

selves multiply. Accordingly, there is a need for systems that 60 The configurable logic includes a user interface for 

simplify management of storage systems, while taking accepting configuration data and a memory that stores tables 

advantage of the flexibility and power of the SAN architec- or lists of the respective sets of driver modules that comprise 

ture, the data paths. The configurable logic in one embodiment is 

„ implemented using a graphical user interface, for example 

SUMMARY OF THE INVENTION „ J, ^j^p^y ^^^^^^f sc«ei. fo, »ciepUng injul 

The present invention provides a storage server architec- signals. The graphical user interface allows for implemen- 

ture supporting virtual devices and virtual circuits for storing tation of configuration tools that are flexible and easy to use. 




ft c 
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According to another aspect of the invention, the con- FIG. 8 is a simplified diagram of a target server internal 

figuration logic includes memory for storing configuration service module according to the present invention, having a 

data in the form of tables that identify the data paths for local answer capability. 

virtual circuits. The memory in one embodiment is imple- piG. 9 is a diagram of a internal service module imple- 
mented using a persistent tables storage process which 5 mcnting a disk mirror. 

maintains the tables in a non-volatile memory that survives • pjQ iq is a diagram of an internal service module 

a reset and/or power down of the storage system. In addidon, implementing a partiUon function. 

the configuration logic implements the data paths for the . ^ - . i j i 

...... - J J. J- ji FIG. 11 IS a diagram of an mtemal service module 

vu^tual circuits usmg redundant driver modules on redundant . , . t. c 

hardware in the system. Thus, tio single point of failure on lO ™Pi^menUng a cache function. 

the storage system will interfere with a particular storage FIG. 12 illustrates a virtual circuit configuration accord- 

transaction. m the present invention. 

The communication protocols supported on the interfaces FIG. 13 is a diagram of a the internal service module 

adapted for communication with users of the data in various implementing a persistent table store manager according to 
embodiments include fiber channel FC protocols such as the 15 the present invention. 

Fiber Channel Arbitrated Loop (FC-AL), and the Fiber FIG. 14 illustrates schematically a persistent storage hard- 
Channel — Small Computer System Interface (SCSI) based ware driver module according to the present invention, 
protocol known as FCP. Other protocols include such pro- pio. 15 illustrates a storage area network having a storage 
tocols as the Internet Protocol (IP), the High Performance server according to the present invention configured as a 
Parallel Interface (HIPPI), the Intelligent Peripheral Inter- 2.0 ^iq^^^q router or a storage director. 

face (IPl) and other protocols coinbined with or in place of illustrates a storage area network in an alternative 

these examples. Accordmg to each of these protocols, stor- configuration having a storage server according to present 

age transacuons can be executed which identify particular ^^^^.^^ configured as a storage router or a storage director 
target storage extents, such as using tiie SCSI target ID and ^ heterogeneous network, 

a Logical Unit Number (LUN). Thus, in this example, a 25 

storage transaction which identifies a SCSI target ID and a . PI^; 1^ ^"^^^"^^^ ^ "^^^^ ^°^Pl«^ ''^^ 

LUN; can be mapped in response to the target ID and the ^^^^"^l^S several storage servers accordmg to the present 

LUN to a particukr virtual circuit iu the storage server ^^^'^"^ ^^^^ du-ect communication chamiels between 

Internal protocols for communicate on among the plural- 
ity of drivers include a variety of block transfer protocols, DETAILED DESCRIPTION 
such as formats similar to the standard 120 block storage 

architecture (BSA) format. Overview 

The present invention provides an intelligent storage , . „. 

routing device which manages logical and physical access to F^G- 1 illustrates a vanety of uses for intelligent storage 

a pool of shared storage devices. The device is logically area network (IS AN) servers. A storage area network (SAN) 

closer to the client server than to the storage devices, and can be used to provide data storage services for chent 

responds to client specific requests for storage transactions computers. A storage area network is optimized to provide 

which require accessing and sharing a storage area network high bandwidth and high throughput storage for chent 

coupledtothedevice.Thedevicemanagestheuseofacache computers such as file servers, web servers and end user 

memory to enhance performance. Key functions provided at computers. An ISAN server, accordmg to the present 

the device include authentication and access permission invention, provides additional functionahty beyond data 

logic, virtual circuit mapping, storage mirror functions, and storage and retrieval such as storage routmg and virtual 

a large non-volatile cache. With tiicsc basic functions, many <^evice management- 
powerful storage management applications can be imple- FIG. 1 includes the servers lOOA-D, the ISAN servers 

mcnted with existing servers and storage devices, which 102A-F, the thin servers 104A-C, and a storage array 106. 

facilitates the addition of new storage to networks. The servers 100 A-D can be UNIX servers, Windows™ NT 

„ ^„^^ servers, NetWare''*' servers or some other type of flic server. 

BRIEF DESCRIPTION OF THE HGURES .aaai^ i^ii * * u 

The servers lOOA-D are coupled to chent computers by 

FIG. 1 illustrates a variety of uses for intelligent storage network links. The ISAN server 102A is coupled to the 

area network servers. server lOOA by a network link. The ISAN server 102A 

FIG. 2 is a block diagram of an intelligent storage area provides data storage services to the server lOOA by per- 

nctwork server. forming the requested storage transactions. The ISAN server 

FIG. 3 is a block diagram of the hardware architecture of 102Ais treated like a storage device by the server lOOA. The 

an intelligent storage area network server. ISAN server 102A is capable of hokling more storage than 

FIG. 4 is a block diagram of the software modules of an a typical hard disk drive or hard drive array. The ISAN 

operating system and supporting programs for an intelligent server 102A can be used as a storage router and provide 

storage area network server. intelligent routing among data stores coupled to the ISAN 

FIG. 5 is a simplified diagram of a hardware driver server 102 A. 
module for a fiber channel interface for use in the system of The ISAN server 102A also provides higher bandwidth 

the present invention. and higher throughput processing of storage transactions 

FIG. 6 is a simplified diagram of a soUd state storage than a typical hard disk drive or hard drive array. The ISAN 

system including a hardware driver module of the present server IdflA can therefore handle the volume of demands 

invention. created by multimedia data streams and other large volume 

FIG. 7 is a diagram of an internal array of disk drives 65 data streams, 
mounted in one embodiment of a storage server according to To provide the highest tiiroughput, the ISAN server 102A 

the present invention. may be coupled to the server lOOA by a high speed network 
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media such as a fibre channel. The servers lOOB-D are 
coupled to client computers by network links. The servers 
lOOB-D arc coupled to a storage area network by a fibre 
channel fabric. The storage area network includes the ISAN 
servers 1«2B-D and the storage array 106. The servers 
lOOB-D and the ISAN servers 102B-D support drivers for 
a fibre channel arbitrated loop (FC-AL). 

Communication between the servers lOOB-D and the 
storage devices over the FC-AL can be accomplished using 
a protocol compliant with the standard small computer 
system interface version 3 (SCSI-3) preferably using a fiber 
channel medium, also termed fibre channel protocol (FCP) 
(e.g., SCSI-X3T10 and FCP X3.269-199X). In other 
embodiments, protocols such as the Internet Protocol are 
used over the fibre channel fabric 108 carrying storage 
transactions in a variety of protocols. In some embodiments, 
the ISAN server 102A supports multiple protocols. 

The thin servers 104A-C are coupled to cUenLs by net- 
work links, but in this example, they are not using storage 
area networks to provide data storage. 

The ISAN servers 102E-F are directly coupled to clients 
by network links. There is no intermediate server. The ISAN 
servers 102E~F may provide application specific processors 
(ASPs) that provide functionality such as file servers, web 
servers, and other types of processing. 

An Intelligent Storage Area Network Server 

FIG. 2 is a block diagram of an ISAN server in one 
preferred embodiment, such as the ISAN server 102 A, or 
some other ISAN server. 

The ISAN server 102A has connection options 130 
including a set of communication interfaces adapted for 
users and for other data processing functions, and storage 
options 128 including a set of communication interfaces 
adapted for storage devices. The ISAN server 102A has a 
hardware interface 126, an operating system 124, a block 
storage interface 118, a management interface 120, and a 
protocol interface 122. The connection options 130 include 
serial connections 140, a front panel connection 142, an 
Ethernet connection 144, and a network interface 146. The 
storage options 128 includes the drive array 132, the solid 
state drive (SSD) 134, the SCSI interface 136, and the 
network interface 138. The SCSI interface 136 is coupled to 
a DVD/CD-R 148. The network interface 138 is coupled to 
an ISAN server 102G and/or storage 150. 

The connection options 130 arc various methods of con- 
necting servers and clients to the ISAN server 102A. The 
serial connections 140 support network management, 
modems for remote management, and uninterruptible power 
supply messages. Hie front panel connection 142 supports a 
management connection with the front panel display of the 
ISAN server 102A. The Ethernet connection 144 supports ao 
Elhemet interface for management protocols and possibly 
for data transfer. The network interface 146 is one of 
potentially many high speed interfaces on the server. In 
some embodiments, the network interface 146 is a fibre 
channel interface with drivers for a fibre channel arbitrated 
loop (FC-AL). The network interface 146 may also include 
drivers for SCSI-3 over the fibre channel medium using fibre 
channel protocol (FCP). 

The hardware interface 126 provides interface specific 
hardware components. For example, the network interface 
146 has a network interface specific set of software modules 
to support configuration, diagnostics, performance 
monitoring, and health and status monitoring. 

The operating system 124, the tables 116, and the inter- 
faces 118-122 support the virtual device and storage routing 



6 

functionality of the ISAN server 1D2A. These components 
of the ISAN server 102A route storage transactions among 
appropriate storage options 128 and the connection options 
130 using configured sets of driver modules in the system. 

5 The operating system 124 provides message routing and 
transport facilities in addition to fail-safe facilities. The 
message routing and transport facilities of the operating 
system 124 are used to route messages, including storage 
transactions, between the components of the ISAN Server 

30 102B. These messages include message in the internal 
format between components of a virtual circuit. These 
messages can also include control messages in other for- 
mats. 

The block storage interface 118 provides software mod- 
15 ules to support block data transfers. The interface 118 
includes support for striped data storage, minrored data 
storage, partitioned data storage, memory cache storage, and 
RAID storage. Tlie different supported storage types can be 
linked to form various combinations such as a mirrored data 
storage with a memory cache. 

The protocol interface 122 provides software modules for 
translating and responding to requests in a variety of pro- 
tocols. One set of modules is provided for the layers of an 
Ethernet connection: the hardware driver, the data link 
^ driver, the Internet protocol (IP) driver, the transmission 
control protocol (TCP) driver, the user datagram protocol 
(UDP) driver, and other drivers. Another set of modules 
provides drivers for FCP. 

The management interface 120 provides software mod- 
ules for managing the ISAN server 102A. The management 
interface 120 contains interfaces for managing access to the 
tables 116. The management interface 120 also contains 
interfaces for rules based management of the system includ- 
ing; scheduhng, or process orchestration; monitoring the 
system; informed consent management; and handling sys- 
tem processes and events. The informed consent manage- 
ment module is premised on providing rules based manage- 
ment suggestions for configuring and maintaining the ISAN 
^„ server 102A. 

40 

Handling Storage Traasactions 

Storage transactions are received over one of the connec- 
tion options 130. Storage transactioas include read and write 

45 requests as well as status inquiries. The requests may be 
block oriented. 

A typical read storage transaction is comprised of the read 
command and addressing information. A write storage trans- 
action is similar lo the read storage tran.saction except the 

50 request includes information about the amount of data to be 
sent and is followed by the data to be written. More 
specifically, using the SCSI-3 protocol, each device has an 
identifier (ID). The machine issuing the request is called the 
initiator and the machine responding to the request is called 

55 the target. In this example, the server lOOA is the initiator 
and has ID 7. In this example, the ISAN server 102 A is the 
target and has ID 6. The SCSI-3 protocol provides for two 
or more addressing components, a logical unit number 
(LUN) and an address. 

60 The LUN specifics a subcomponent of the target ID. For 
example, in a combined hard disk/tape drive enclosure, the 
two devices might share an ID, but have different LUNs. The 
third addressing component is the address where the device 
data is to be read from or stored to. The ISAN server 102A 

65 provides for virtual LUNs on a per initiator basis. Thus a 
single ISAN server 102A might support, for example, ten 
thousand virtual LUNs or more. 
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The ISAN server 102A will map the SCSI-3 storage provide new functionality and easier administration by sup- 
transaction request to a virtual circuit corresponding to a porting storage routing. 

virtual LUN. A virtual circuit is a sequence of one or more In the original configuration, the server lOOA provides 

virtual devices. A virtual device is a composed of one or data backup and archiving functionality. The backup and 

more devices such as software modules or hardware com- 5 archiving funaions tic up the processor of the server lOOA 

ponents. For example, two network interface devices could with important but bandwidth intensive funcUons. In the 

be combined to be a virtual device. Similarly, two cache configuration of FIG. 2, the ISAN server 102A acts as a 

devices could be combined as a virtual device. This design storage roister for storage irarisactions requested by the 

permits components to fail without disrupting the storage ^^^7" l/^A. The ISAN server K^Acan then direct backups 

transactionproccssingcapabiIitiesoftheISANserverl02A. lO and archivmg without tymg up the server. 

. . -.ij-. The ISAN server 102A provides another advantage m 

A virtual circuit composes the necessary virtual devices to ^ ^ ^ ^ ^^^-^^ j ^ 

support a storage Iraasaction. Typically, the first component ^ ^ ^^^^^ ^^^^ ^ ^^.^^ ^s 

in the virtual circuit IS a driver for translation of the storage ^^^^^^^ terabytes of storage is being brought onto the 

transaction from the storage transaction communication network, the ISAN server 102 A can migrate the data to the 

channel format, FCP in this example, to an mtemal format. 15 ^^^^ ^^^^^ consuming processor time on the server 

One such internal format may be similar to the mtelligenl ^qq^ p^^her, the mirroring drivers support intelligent read 

input and output (1,0) block storage architecture (BSA) ^^ite functionality that allow the storage 150 to be 

message format. The internal format is storage medium and gj^f^^^j the new drive array without disrupting access to 

communication channel neutral in the preferred system. jj^^ ^^^^ 

The intermediate virtual devices of a virtual circuit pro- ^° -j^us, as data is moved to the new storage array, read and 

vide additional services such as caching, mirroring, RAID, write requests for those portions of the data will be directed 

etc. Because the internal formal is storage medium neutral, iq new array while requests for other data portions will 

all of the inteniicdiatc virtual devices can be designed to be directed to the old storage 150. Once the migration is 

operate on the internal format and thus interoperate with complete, the virtual circuit can be adjusted to remove the 

other virtual devices in the circuit. storage 150 from the virtual circuit. 

The final virtual device in a virtual circuit is typically the The ISAN server 102A, in one embodiment, provides a 

format translation and communication channel drivers for platform for cache memory which is shared among a plu- 

controlling the storage. For example, the drive array 132 is rahty of virtual circuits. 

controlled by redundant hardware driver modules (HDMs) Because a storage transaction is not linked to a data store, 

that are grouped to form a virtual device. The HDMs provide ^ possible to change the data stores supporting a virtual 

BSA to SCSI translation and the HDM handles the interface circuit as storage needs change or as different storage 

to the drives that compose the drive array 132. Similarly, if equipment is used, 
the virtual circuit is a link to some other type of storage over 

the network interface 138, there will be a virtual device with Hardware Architecture Overview 

support for BSA translation to the storage device commu- FIG. 3 is a block diagram of one suitable hardware 

nication channel protocol. architecture of an intelligent storage area network (ISAN) 

Hie storage server also includes resources in the operating server. The hardware architecture implements redundancy 

system and at the interfaces to the client servers which and supports distributed software systems to prevent any one 

emulate physical storage devices. The emulation allows the 4q single point of failure from interfering with a particular 

virtual devices to appear to the client servers accessing the storage transaction. 

storage as if they were physical devices. Thus, the chent FIG. 3 includes the ISAN server 102A. The ISAN server 

servers can be configured to communicate using standard is designed to provide a high degree of redundancy while 

protocols, such as FCP using SCSI commands for storage using standard components and the standard based devices, 

transactions. In the embodiment utilizing SCSI commands, 45 For example, the ISAN server 102 A uses a high speed 

the emulation involves responding to an inquiry command version of the standard peripheral component interconnect 

according to the SCSI protocol with device identifiers and (PCI) implementation and standard fibre channel arbitrated 

device capability information expected by, or compatible loop (FC-AL) interfaces. A variety of other protocols and 

with, the mitiating server. Also, a read capacity command interfaces can be used in other embodiments, 

and a mode page data command in the SCSI protocol are 50 The ISAN server 102A has four separate 64-bit 66 MHz 

handled by the emulation resources in a manner that allows PCI busses 200A-D. Many dififerent configurations of stor- 

the client servers using the storage to rely on standard age devices and network interfaces in the slots of the PCI 

configuration information for physical storage devices, busses are possible. In one embodiment, the PCI busses are 

while the storage server spools the client server by emulat- divided into two groups: the SSD PCI busses 200A-B and 

ing the physical storage devices at the interface with the 55 the interface PCI busses 200C-D. Each group has two 

client server, and maps actual storage transactions to virtual busses that are designated by the terms upper and lower. The 

devices. The emulation resources also allow the virtual upper and lower busses in each group can be configured to 

devices to be identified by the combination of an initiator, a provide redundant services. For example, the lower SSD 

logical unit number LUN, and a target device identifier, PCI bus 200B has the same configuration as the upper SSD 

without requiring the storage transaction to be tied to the PCI bus 200 A. 

specific physical target device identified in the request. The PCI busses 200 A-D are connected to host bridge 

controller (HBC) modules 202A-B. The HBC modules 

Storage Routing 202A-B span the PCI busses 200A-D and provide redun- 

The ISAN server 102B supports storage routing. For dant bridging paths, 

example, if the server lOOA is originally coupled to the 65 The SSD PCI busses 200A-B support solid state drive 

storage 150 in a SAN, the ISAN server 102B can be added (SSD) modules 204A-G. The SSD modules 204A-G pro- 

to the SAN between the server lOOA and the storage 150 to vide solid state storage devices such as a flash memory store. 
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The interface PCI busses provide an interconnection from 
the network interface controller (NIC) modules 206A-B, the 
redundant arrays of independent disks (RAID) Controller 
(RAG) modules 212A-B, and application specific process- 
ing (ASP) modules 208A-D to the HBC modules 202A-B. 

In addition to coupling the ISAN server 102A to the 
external FC-AL, the NICs 206A-B can be coupled to fibre 
channel hub (FCH) modules 214A-D. Each FCH module 
214A-D is coupled to both NIC modules 206A-B. Each 
FCH module 214A-D provides ten FC-AL ports, and can be 
cascaded through the NIC modules 206A-B to provide a 
twenty station FC-AL hub. 

The disk drive hub (DDH) modules 216A-D provide a 
redundant FC-AL fabric to connect disk drives to the RAC 
modules 212A-B. The FC-AL fabric in each of the DDH 
modules 216A-D comprises two redundant loops, which 
couple all of the drives attached to the DDH module with 
both RAC modules 212A-B. The RAC modules manage a 
loop among all of the DDH modules 216A-D. The DDH 
modules 216A-D each support five dual-ported disk drives 
such as the disk drive 218. 

The system mid -plane (SMP) is not depicted in FIG. 3. 
The SMP is a passive raid-plane that provides the intercon- 
nections shown in FIG. 3 between the HBC module 
202A-B, the SSD modules 204A-H, the RAC modules 
212A-B, the NIC modules 206A-B, the FCH modules 25 
214A-D, the DDH modules 216A-D, and the ASP modules 
208A-D. The SMP ls compact PCI based, with four custom 
compact PCI busses 200A-D, RAC-DDH interconnections, 
and NIC-FCH interconnections and miscellaneous control 
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Redundancy and Fail-Over 

The ISAN server 102A offers a high degree of redun- 
dancy. In one embodiment, there are redundant NIC, RAC, 
and HBC modules. The SSD modules and drives support 
mirroring. The drives also support parity and dual channel 
access. Each DDH module contains a fully redundant 
FC-AL fabric for connection to the RAC modules. Fail-over 
is handled by the HBC modules, which control the other 
modules in the ISAN server. The control is multi-layered. 

The HBC module's first layer of control-over is the power 
supply control. Each module has an individual power supply 
enable signal that is controlled by the CMB controller on the 
module. Although the HBC modules are redundant, only one 
HBC module will serve as the master HBC module and 
direct and control the system. The other HBC will act as a 
slave. When a module is plugged into a slot, its power 
.supply is initially disabled. Only the master HBC module 
can enable the power supply. If a module begins to perform 
improperly and does not respond to commands, the HBC 
module can disable the power supply to the module. 

The second layer of control for the HBC modules is the 
card management bus (CMB). Each module has an Atmel 
AT90S8515 (AVR) microcontroller that is coupled to the 
CMB. The HBC module itself has an AVR microcontroller 
316 coupled to the CMB that can act as a master or as a 
slave. The CMB microcontroller is powered by a connection 
to the midplane independently of power supplied to the main 
processor on the module. The CMB allows the master HBC 



busses comprising the mid-plane signals. In addition, the 30 ^^^^ ^ ^yp^> determine if a card is present, send a 



SMP provides power distribution from the power subsystem, 
not depicted in FIG. 3, to the modules, at voltages of 48V, 
12V, 5V, and 3.3V. 
The front panel display (FPD) 220 provides a user inter- 



non-maskable intermpt to a card or perform a hard reset of 
a card. Module processors and the master HBC module can 
also perform communication through a serial port on the 
AVR microcontroller on the module. This communication 



face for the ISAN server 102A. The FPD contains a display 35 path can be used as a backup for control communication in 
device and an input device. In one embodiment, a (ouch *' " "™ 

sensitive liquid crystal display (LCD) is used to present a 
touch sensitive screen with input capabilities. The FPD 220 



is coupled to the HBC modules 202A-B to support status 



(he event of a PCI failure. 

The third level of control for the HBC modules is the PCI 
bus. If a module does not respond using a control process on 
the PCI bus, it can be queried via the CMB. If the module 



displays, configuration display and management, and other gjin ^oes not respond, a non-maskable interrupt can be set 



management functions. 

Power and fan subsystems, not depicted in FIG. 3, provide 
redundant AC to DC power supplies, redundant DC to DC 
power conversion, battery backup for power down, and a 
redundant push-pull fan subsystem. These components sup- 45 
port the high availability and low down time features that are 
important when a storage area network is deployed. 

The ISAN server 102A can be coupled to other ISAN 
servers to appear as a single network port in a storage area 



via the CMB. If the module still does not respond, it can be 
reset via the CMB. If after reset, the module still does not 
respond, it can be powered down and a warning can be 
issued to replace the module. 

Software Architecture Overview 

An ISAN server is supported by an operating system that 
is designed to support the uniquely high bandwidth, high 
throughput, and demands of a storage server. The operating 



network or as a network attached storage device. This 50 systems schedules and controls data transfers over the bus 



coupling can be done over the FC-AL expansion ports that 
are coupled to each of the HBC modules 202A-B. 
Additionally, the HBC modules 202A-B offer RS232 serial 
ports and 10/100 Ethemet ports for out-of-band manage- 
ment. 

The bus system includes all of the buses in the ISAN 
server 102 A. In this example, the bus syslcm includes the 
four PCI buses interconnected by the host bridge controllers. 
The bus system also includes the PCI buses internal to the 
HBC modules that provide additional interfaces. The slots 
include all of the positions on the bus system which can 
receive interfaces. In this example, each of the four PCI 
buses outside of the HBC modules can accommodate four 
interfaces. 



55 



(SO 



systems and manages the system. Although a number of 
different operating system and software component struc- 
tures are possible, in one embodiment, a highly modular 
operating system designed for a storage server is used. 

FIG. 4 is a block diagram of the software modules of an 
operating system and supporting programs for an ISAN 
server. 

FIG. 4 includes the following operating system compo- 
nents: the hardware interface module 900, the Nucleus 
PLUS™ real-time kernel module 902 available from Accel- 
erated Technologies, Inc., Mobile, Ala., the ISOS protocol 
management module 904, and the storage services module 
906. The hardware interface module 900 allows the software 
components of the ISAN server to communicate with the 



The interfaces are the cards or other devices that are 65 hardware components of the ISAN server, 
placed in the slots. The interfaces support drivers and The Nucleus PLUS™ real-time kernel module 902 is used 
hardware for the data stores coupled to the interfaces. to provide basic operating system functions such as: tasks, 
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queues, semaphores, tuners, and critical section support. The 
Nucleus PLUS'T" real-time kernel module 902 is exported to 
the software modules of the ISAN server as functions in C++ 
classcs by the storage services module 906. 

The ISOS module 904 permits the ISAN server to support 
a messaging architecture for input and output. The hardware 
modules such as the RAID controller (RAC) modules, the 
network interface controller (NIC) modules, the solid state 
drive (SSD) modules, the disk drive hub (DDH) modules. 



ferred architecture of the present invention. FIG. 12 provides 
a simplified diagram of a set of driver modules which have 
been configured into data paths acting as virtual circuits. 

FIG. 5 illustrates a network interface card 500 having a 
HDM 504. The card 500 has a physical interface 501 to a 
fiber channel network. A network interface chip 502, in this 
example a Qlogic device, such as a ISP 2200A provided by 
Qlogic Corporation of Costa Mesa, Calif., is coupled to the 
physical interface 501. The network interface chip 502 



and the fibre channel hub (FCH) modules, are all input/ lO generates communication represented by line 503, which is 

output processors (lOPs). The master host bridge processor processed in the HDM 504. The HDM 504 conditions the 

(HBC) module serves as the host. communications for use by other driver modules in the 

The storage services module 906 implements messaging system. Thus, communication represented by Une 505 has an 

classes to support the reliable Uansport of messages between SCSI fomiat. Communication represented by line 506 has an 

components. The storage services module 906 supports the message format such as a BSA format. Communication 

operation of device driver modules and support for virtual represented by line 507 has an Internet Protocol (IP) format. 



devices. The device driver modules (DDMs) and virtual 
devices (VDs) are the building blocks of the ISAN server 
storage system. The storage services module 906 is orga- 



nized around providing support for requests for storage #1. 



The HDM is an instance of a driver class labeled "QLogic 
driver" in the diagram, and given device identifier DID 401 
in this example. The physical interface is identified as NIC 



transactions. 

In some applications, a single ISAN server such as the 
ISAN server 102 A will have several hundred DDMs oper- 
ating in conjunction with the operating system modules 
900-906 to support responses to storage server requests. 
Other applications use a few DDMs in various combina- 
tions. 

Software components are implemented as device driver 
modules (DDMs). A DDM that primarily services requests 
for a hardware device is termed a hardware driver module 
(HDM). A DDM that serves as an internal, intermediate 
program is termed an intermediate service module (ISM). 
For example, the DDMs that service the SSD modules are 
termed HDMs. The DDMs that provide cache services, 
mirroring services, and other types of services not directly 
linked to a hardware device could be termed ISMs. 

A single DDM can have multiple instantiations on a single 
ISAN server. For example, in FIG. 4, there arc four instan- 
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FIG. 6 illustrates a storage device 700 which is imple- 
mented by an array of nonvolatile integrated circuit memory 
devices. The preferred embodiment of the device 700 is 
described in co-pending U.S. patent application Ser. No. 
09/292,536, cntiUcd High Speed Bus Interface for Non- 
Volatile Integrated Circuit memory Supporting Continuous 
Tranter, filed Apr. 15, 1999, which is owned by the same 
assignee as the present application, and is incorporated by 
reference as if fully .set forth herein. The HDM 702 is 
coupled with the array 701, and translates the block storage 
architecture communications on line 703 into format for 
storage and retrieval from the array 701. In this example, the 
HDM 702 is given a device identifier 1130. The physical 
interface is identified as SSD #4. 

FIG. 7 illustrates the configuration of an array 800 of disk 
drives which are mounted on the storage server chassis in a 
fiber channel arbitrated loop architecture in the preferred 
embodiment shown in FIG. 3. Fiber channel disk hub #0 



tiations of the performance, health and status PHS monitor 40 216A, channel disk hub #1 216B, fiber channel disk of n 

908A-D, one for each of the four major software sub- 216C, and fiber channel disk hub #3 21 6D, which are also 

systems: the NIC 910, the RAC 920, the HBC 930, and the illustrated in FIG. 3, are coupled to redundant hub control 

SSD 940. Each DDM has its own message queue and a HDMs 801 and 802. 

unique identifier. For example, the PHS monitor 908Aon the HDMs 801 and 802 are connected with physical fiber 

NIC 910 might be device id (DID) 0. Each DDM also lists 45 channel arbitrated loop connections 803 and 804 respec- 

the class of storage requests handled by the DDM and the tively. The HDM 801 is given device identifier 1612 and the 

operating system modules route the requests to the DDMs HDM 802 is given device identifier 1613. The connection 

based on the class of the storage request. Requests can be 803 is coupled to a fiber channel interface 805. Interface 805 

routed by request codes or by virtual device numbers. includes a network interface chip 806 which is coupled with 

The NIC software subsystem 910 includes three DDMs: so physical interface 820, and to a HDM 807. An ISM 808 is 

a processor support HDM 912A, an input/output translation coupled to the HDM 807 and to the internal communication 

ISM 914A and the PHS monitor 908 A. The RAC software path 809. The ISM 808 translates the block storage archi- 

subsystcm 920 includes three DDMs: a processor support lecture communications on line 809 into lOCB communi- 

HDM 912B, an input/output translation ISM 914B, and a cations for the HDM 807. The HDM 807 communicates 

PHS monitor 908B. The HBC software subsystem 930 55 with the network interface chip 806, which in turn drives the 

includes: a processor support HDM 912C, an input/output fiber channel 803. The ISM 808 is given device identifier 

translation ISM 914C, a card management HDM 916, a 1210, and the HDM 807 is given device identifier 1110. The 

system monitor DDM 918, an Internet Protocol DDM 921, physical interface 805 is labeled RAC #0. 
a front panel display DDM 922, an application specific The fiber channel connection 804 is coupled to interface 

processor support DDM 924, and a PHS monitor 908C. The 60 810. Interface 810 has a configuration like interface 805. 

SSD software subsystem 926 includes a solid state drive Thus the interface 810 includes a physical fiber channel 

management HDM 926 and a PHS monitor 908D. The front interface 811 which is driven by nclwork interface chip 812. 

panel display 950 supports a hypertext markup language The network interface chip 812 communicates on the chan- 

(HTML) client 928. ncl represented by line 813 with HDM 814. HDM 814 

FIGS. 5-7 illustrate a variety of hardware driver modules 65 communicates with ISM 815 via channel 816. The ISM 815 

HDMs and FIGS. 8-11 illustrate a variety of internal, manages an interface to the BSA formal messages on 

intermediate service modules ISMs according to the pre- channel 817. In this example, the ISM 815 is given device 
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identifier 1211. The HDM 814 is given device identifier 
1111. The interface 810 is identified as RAC #1. 

FIGS. 5-7 illustrate a variety of physical coramunication 
interfaces and corresponding HDMs. FIGS. 8-11 illustrate a 
variety of ISM examples according to the present invention, 
which can be configured into data paths. 

FIG. 8 shows a SCSI target server 550, which is one 
example of a protocol server module according to the 
present invention. Similar protocol server modules can be 



drive interface 756, The partition logic process 753 config- 
ures the subject storage device identified by the drive 
process 756, using a logical partitioning function useful for 
a variety of storage management techniques, so that the 
physical device appears as more than one logical device in 
the virtual circuits. In this example, the partition ISM 750 is 
an instance of a class "partilion," and given device identifier 
10400. 

FIG. 11 illustrates a cache ISM 850. The cache ISM 850 



implemented for any particular storage channel or network includes logic processes 853 which communicate with an 



protocol implemented by users of the data managed through 
the storage server of the present invention. The target server 

550 has a message interface 551 which receives incoming 
messages from an HDM, such as the HDM of FIG. 5, 
coupled to a communication interface adapted for connec- 
tion with a user. In this example, the messages on interface 

551 have an SCSI format. In other examples, the messages 
may already have the BSA architecture, or some other 
architecture which is suitable for the protocol on the com- 



interface 851 to the internal message passing structure on the 
storage server. Data structures in the cache ISM 850 include 
a local cache memory allocation 854, a cache table 855 
which identifies the data stared in the cache 854, and a drive 
15 interface 856. The drive interface communicates on channel 
857 with an HDM associated with the particular virtual 
circuit being served by the cache. The cache memory 854 in 
one embodiment is managed locally in the storage server. In 
an alternative embodiment, the cache can be stored in a high 



munication interface being served. The server 550 includes speed non-volatile memory, such as a soUd state memory 
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a switch function 550 which translates incoming messages 
to a SCSI to BSA translator 553, or to an answer local 
function 554. Typically messages arc forwarded by the 
translator 553 as outgoing messages on fine 555. Incoming 
messages on line 555 are supplied to translator 556 which 
translates the incoming BSA messages to the SCSI format 
used on line 551. 

In many instances, the SCSI target device can respond 
using the local answer service 554 to the SCSI message 
without routing the message further. Many status messages 
that do not relate to reading or writing from the storage itself 
arc handled by the local answer service 554. 

The target server 550 in this example is an instance of a 
class SCSI target server, and given a device identifier 500. 35 
One function of the protocol server, such as the SCSI target 
server 550, is to identify the storage extent which is subject 
of a storage transaction on the associated interface. The 



module having architecture like that described with respect 
to FIG. 6. In the preferred embodiment, the cache ISM 850 
is implemented as an instance of a class "cache," and given 
a device identifier 10300. 

FIG. 12 provides a heuristic diagram of redundant virtual 
circuits implemented by data paths including a plurality of 
driver modules according to present invention. Virtual cir- 
cuits include an external interface for communication with a 
user of the data, a protocol translator for translating com- 
munications with the user into the communication format of 
the driver modules, and a storage object which includes a 
communication interface to a storage device. Storage opera- 
tors which perform data path tasks can exist between the 
translator and the storage object. The optimal ordering of the 
driver modules acting as storage operators, such as cache, 
mirror, partition, etc., is done by the system designer using 
the configurable logic provided by the storage server. 
In the example illustrated in FIG. 12, the external inter- 



storage extent is mapped to a virtual circuit using the 

configurable logic in the storage server as described in more face is provided by the NIC #0, and its associated HDM is 

detail below. represented by block 1010. The protocol translator is pro* 

FIG. 9 illustrates an ISM 650 which performs a mirror ^ided by the SCSI target server ISM 1011. A cache function 

management data path task. The ISM 650 includes an is provided by the ISM 1012. A mirror function is provided 

interface 651 which is connected to the internal communi- by the ISM 1013. The storage objects are accessed from the 

cation channels on the device. Logic processes 652 receive 45 mu-ror function 1013, and consist of a set of physical storage 

the incoming communications and data and manage a mir- interfaces selected m this example from the fiber channel 

roring function. The logic 652 communicates with a plural- basic daisy cham interface and its assoaated HDM repre- 

ity of drive interfaces including primary drive 653, second- ^^n^ed by block 1014 or an extemal LUN interface, the disk 

ary drive 654, tertiary drive 655, and standby drive 656. t^n^cs in the fiber channel arbitrated loop accessed through 

Although 3-way mirroring is shown in the diagram, any 50 '^^ ISM/HDM pair represented by block 1015 and the 

number of mirror paths may be implemented for "n-way" redundant block 1016. the solid state storage device and its 

mirrors using virtual circuits. Although the drive interfaces associated HDM represented by block 1017, and the rater- 

in FIG. 9 are labeled with the term "drive," other types of face lo an external disk drive and its associated ISM/HDM 

storage devices can be used in the mirroring functions. The paif represented by block 1018. Separate HDM modules on 

drive interfaces 653-656 communicate using the internal s5 each of the fiber channels interfaces to disks (01), (02), (03), 

communication channels with the HDM modules associated ^nd (04) manage the communication across the fiber channel 

with the target storage devices used in the mirroring arbitrated loops with the interfaces 1015 and 1016. 

function, or with other ISM modules as suits the particular In the embodiment shown, the mirror module 1013 

virtual circuit. In this example, the mirror ISM 650 is accesses disks (01), (02), and (04) as the primary, secondary 

implemented as an instance of a class "mirror," and given a 60 standby drives, respectively, for the mirror functions. 



device identifier 10200. 

FIG. 10 illustrates a partition ISM 750. The partition ISM 
750 includes an interface 751 which receives internal com- 
munications from other driver modules, and an interface 752 
which also communicates with other driver modules. The 
ISM 750 includes logic processes 753, data structures for 
storing a base address 754 and a limit address 755, and a 



65 



Although the mirror module shown in FIG. 9 includes the 
tertiary drive interface, this tertiary drive is not used in the 
example system. 

Also shown in the diagram are partition ISM modules 
1020 and 1021, which are not connected with the data paths 
of the virtual circuit shown. These blocks are present to 
illustrate that using the virtual circuit structure, new 
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modules, like partitioning, may be added to the path by 
simply configuring the storage server. 

A redundant data path is implemented using the interface 
NIC #1 and its associated HDM represented by block 1025, 
the SCSI target server ISM represented by block 1026, the 
cache ISM represented by block 1027, and the mirror ISM 
represented by block 1028. Redundancy in the data storage 
devices is accomplished using the mirror function. The 
redundant driver modules are distributed in a preferred 
embodiment on separate lOPs within the storage server. 

As illustrated in FIG. 12, each of the driver modules 
includes a unique driver identifier which Ls shown within the 
parentheses in the blocks of FIG. 13. The unique device 



cults. The external LUN table 1413 identifies logical units of 
storage which are maintained in other storage servers con- 
nected through the external storage interface on the storage 
server. 

Two primary tables support the exporting of storage to 
clients and the storage routing functionality of the IS AN 
.server 102A. These tables are the export table 1407 and (he 
virtual device configuration table 1410. 
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The Export Table 1407 

The export table 1407 maps addressing information 
received with a storage transaction to a virtual circuit or to 
a storage option. In the case of SCSI-3 over a fibre channel 



identifiers are used to support the configuration logic based ^^^'^S" 7""", " ""r,."""',"' 

.... e A * w _ Au..tu ^t^^ „^ 15 mterface, the addressmg mformation used IS the imtiator ID, 

on tables m a configuration database managed by the storage * , ^ j -u . . j j 

" the target LUN, and the target address. 
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It is not necessary to use all of this information to resolve 
each request because many LUNs can be shared across all 
initiators, or clients, and most LUNs will use the target 
address, e.g., the ofifeet on the storage device, for addressing 
within the virtual circuit rather than for selecting different 
virtual circuits. Thus in a typical embodiment, the export 
table 1407 is organized as shown in Table 1. 

TABLE 1 







Imtiator 


Fi/st virtual 


Primary 




Protocol Specific 


Specific? if 


device in 


connection 


Protocol 


Addressing (LU^f) 


yes, ID 


ciiuiiit 


owner 


SCSI 


0 


No 


11 


NICO 


SCSI 


1 


Yes. ID - 6 


30 


NICO 


SCSI 


1 


Yes. ID - 5 


60 


NIC 1 


SCSI 


2 


No 


12 


NICO 


TCP/IP 


Port 2000 


No 


70 


NICO 



server, and controlled by local configurable logic in the 
storage server. 

In the preferred system, the configuration tables are 
managed by a persistent table driver, such as that illustrated 
in FIGS. 13 and 14. Referring back to FIG. 2, the ISAN 
server 102 A stores management and routing information in 
tables such as the tables 116. The tables 116 can be accessed 
through the management interface 120. The tables 116 will 
typically be stored in persistent memory such as a non- 
volatile memory. The tables 116 can be maintained redun- 
dantly to provide fail safe support. 

HG. 13 illustrates a persistent table module 1400 which 
is implemented as an uistance of a class "persistent table," 
following the basic architecture of the driver module struc- 3Q 
ture. The persistent table module 1400 includes a table 
access logical processor 1401, and a variety of supporting 
functions including table data access manager 1402, persis- 
tent image manager 1403, and persistent table instance 
synchronization module 1404. The table data access man- 35 

ager 1402 is coupled with a table class manager 1405 in this -j^^ ^j^^qj^ table 1407 may include other columns such as 
embodiment. The table class manager manages a plurality of ^^^^ current state of the virtual circuit, the capacity of the 
configuration tables including a fiber channel port ID table ^^^^^^1 circuit, and other information. In one embodiment, 
1406, a LUN export table 1407, a configuration template the export table 1407 lists the entire virtual circuit in a 
table 1408, a DDM roll call table 1409, a virtual device table 40 column of the export table. 

1410, a storaee roll call table 1411, a fiber channel disk roll . , , . .l . . 1 • e 

iw u. 1 * 1 T TIM * Li 1 ^1 :i n .t,f^ Table 1 shows that protocol specific addressmg informa- 

call table 1412, an external LUN table 1413 and a solid state j , , ,1, , *u„ * 

* 1,1 AA^A Tu ,^v,.i„™fi™,..*;«««f(t,^«^f,^f tion can be used to route the request to the appropnatc 

storage table 1414. The particular configuration ol the set 01 , - -ru L.Tr^oV^^^.-.^^ mnn 

.1.1 J u *i- ■ * * * ui -,^.,1= i^nn „„„ virtual circuit. Therefore, only TCP sessions using port 2000 

tables managed by the persistent table module 1400 can be ' > ^^^^^^ storage would be 

changed to suit the parucularimplementauon, and optimized ^^^^^^ ^.^^^^ circuit starting with the virtual device 
for certain classes of devices. . . . . * 

, , . havmg identifier 70. 

The persistent image manager 1403 and the table instance ^ , ^ 

synchronization manager 1404 communicate with persistent Table 1 shows that a smgle LUN for a protocol can be 
data storage driver 1420 as illustrated in FIG. 15, and a connected to different devices depending on the iniUator of 
second persistent storage driver not shown. The persistent 50 ^^^^'^e* transaction. In this example, LUN 1 'S mapped 
data storage driver 1420 is implemented as an HDM, which 
is an instance of a class "persistent storage," and is given a 
device identifier following the model of the driver modules 
described above. In the preferred system, the persistent data 
storage HDM 1420 communicates with the solid slate stor- 55 
age device in the storage server, and provides fast access to 
the data used in the virtual circuits. 

The persistent data storage maintains a wide variety 
configuration information for the system. The DDM roll call 
tabic 1409 includes a list of all the instances of the device 60 
driver modules, and their unique device IDs. The storage roll 
call tabic 1411 includes a list of all the active storage devices 
detected by the storage server. The roll call tables can be 
utilized by the virtual device table 1410 and by the configu- 



to different virtual circuits based on the initiator ID. Also, 
virtual circuits can be mapped based on other types of 
identifiers, such as the World Wide Name WWN. 
An example export table has the following structure: 
#define EXPORT_TABLE "Export_Table" 



struct Exportliblc Entry { 



lowID 
U32 

1)32 



CTProtocolTypc 
U32 

ration tools to create virtual circuits. The LUN export table 65 vdn 
1407 provides a technique for mapping the identified storage 
extents within a storage channel transaction to virtual cir- 



ricfrhisRow; 
version; 

size; 

PfOlocolType; 
CiiaiitNumber; 
vdNexu 



//rowID of this tabic row. 
//Version of Export Thblc 
record. 

//Size of Export 'ftblc 
record in bytes. 
//FCP, IP, other 
//LUN or other 
//Rrst Vulual Device 
number in the Path 
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-continued 
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VDN 


vdLcgBcyBsa; 


//Virtual Device aumbei 






of the legacy BSA 


VDN 


vdLcgacyScsi; 


//Viitual Device Dumbci 






of ihe legacy SCSI 


U32 


BqjortcdUJN; 


//LUN number exported 


U32 


[aiiiaioild; 


//Host ID 


U32 


T^rgcUd 


//ourlD 


U 




//FC Ijoop niunber 


Stiing32 


SerialNumbei; 


//Use a string array for 






Serial Number 


long long 


Capacity; 


//Capaciiy of this Mrtual 






Circuit 


U32 


FailSute; 




U32 


PrimaryFCI^rgctOwner; 




U32 


SccondaiyFCTirgetOwnci; 




CTReadyState 


ReadyStatc; 


//Current state 


CTReadyState 


DesiiedReadyState; 


//Desired Ready State 


Stringie 


WWNName; 


//World wide Name (64 






or 128-bit IEEE 






registered) 


Stiiiig32 


Name; 


//Virtual Circuit Name 


#endif 







The Virtual Device Configuration Table 



The status column indicates the status of the software or 
hardware modules supporting the virtual device. For 
example, in the first entry in T^ble 2, the status is "primary", 
which means that the primary device driver, 4000 here, is 

5 being used. In the second entry in Table 2, the status is 
"alternate", which means that the primary device driver has 
failed or is not responding properly. In that case, the alter- 
nate driver, 1211 for the second entry in Table 2, is used, [f 
a device has more than one alternate, the status column will 

10 indicate the driver being used. 

Example 

For example, consider a storage transaction that comes 
over one of the connection options 130 to the IS AN server 
IS 102Ausing the SCSI protocol and designating LUN 2 in the 
addressing information. Assume that the ISAN server 102A 
is configured as shown in Tables 1 and 2 for this example. 

The connection option such as the network interface 146 
over which the storage transaction is received is coupled to 
a hardware device driver. The hardware device driver 
receives the storage transaction and depending on the 
protocol, dispatches it to an appropriate virtual device for 
handling that protocol. 
For example, SCSI storage transactions are sent to a 



devices with the device drivers that support the virtual 
device. The virtual devices are designed to support a redun- 
dant design. Therefore the table for virtual device configu- 
rations maps virtual device numbers to device modules. In 
one embodiment, a table such as Table 2 is used to map 30 
virtual devices to supporting device drivers. FIG. 12 illus- 
trates the virtual circuit implemented by Table 2, starting 
with virtual device 12. 



TABLE 2 



Virtual 




Alter- 








Device 


Primary 


nates 


Parameters 


Status 


Class 


1 


4000 


4001 


N/A 


Primary 


Peraistent T>iblc 


10 


1210 


1211 


SO(00) 


Alternate 


FC Diak 


11 


500 


501 


VD(IO) 


Primary 


SCSI Target 


12 


500 


501 


VDa3) 


Primary 


SCSI Target 


13 


10300 


10301 


VD(14) 


Primary 


Cache 


14 


10200 


10201 


VDC15.16. 


Primary 


Mirror 








null.l?) 






15 


1210 


1211 


SO(02) 


Primary 


FCDifik 


16 


1210 


1211 


SO(03) 


Primary 


FCDisk 


17 


1210 


1211 


SO(04) 


Primary 


FC Disk 



35 



40 



The virtual device configuration^ table connects virtual 25 ^^^.^^ ^^^^^ g^^gj ^^^g^^ ^^^^ Similarly, IP storage 

transactions are sent to a device driver in the IP target class. 
Here, the storage transaction was made using the SCSI 
communication protocol so it is routed to a SCSI target 
device driver (DID 500). 

The SCSI target device driver further analyzes the 
request. The first part of the analysis is to determine which 
virtual circuit to map the request to. This determination can 
be made using the information in the export table. In this 
example, Table 1, indicates that a request using the SCSI 
protocol specifying LUN 2 should be routed to the virtual 
circuit starting with the virtual device 12. In one 
embodiment, all of the SCSI target requests arc routed to the 
same SCSI target driver for a single interface. In this 
embodiment, the parameter information for the target VD 12 
is used to control the behavior of the SCSI target device 
rather than routing the message to a second virtual device for 
a SCSI target. 

The SCSI target device here, driver number 500, trans- 
45 lates the SCSI message into an internal format. One such 
format is based on the LO block storage architecture (BSA) 
format. This format is device and protocol neutral and can be 
used by the intermediate device drivers. Once the request is 
As T^ble 2 shows, for each virtual device, information is in internal format, it is sent to the next virtual device in the 
provided about primary and alternate driver modules sup- so virtual circuit as indicated by the parameter field, here, the 
porting the virtual device. For example, in the second entry parameter is VD(13) or virtual device 13. 
in Table 2, a fibre channel disk drive is mapped to virtual The message is routed to the VD 13, which provides 

device (VD) 10. redundant caching drivers, hero, the drivers numbered 10300 

The virtual device comprises the one or more software or and 10301. The caching driver uses a memory to cache 
hardware modules for supporting the virtual device. The 55 storage transactions. Based on the caching algorithm being 
parameters column is used to provide initialization informa- 
tion. In the case of VD 10, the parameter is SO(OO) which 
stands for storage option 0. Each device driver module class 
has class specific parameters. Storage option drivers use 
parameters specifying a particular storage unit. Intermediate 60 
driver classes such as the mirror driver and the cache driver 
use parameters that specify the next virtual devices in the 
virtual circuit. This format allows a single device driver 
module to support multiple devices based on the parameter 
setting. Notice that in Table 2, the device driver 1210 is 65 This mirroring driver supports a primary, secondary and 
being used by virtual devices 10, 15, 16, and 17, but each tertiary store as well as a standby store. Other mirroring 
specifies a different parameter to the driver. drivers may support dififerent algorithms. This mirroring 



used by the driver, the driver will route storage transactions 
to the next virtual device in the virtual circuit at appropriate 
intervals. Here that next device is indicated by the parameter 
VD(14), or virtual device 14. 

In the internal format, the message is routed to VD 14. 
The virtual device 14 includes redundant mirroring drivers. 
In this case, the drivers 10200 and 10201 arc used. The 
mirroring drivers implement a mirroring algorithm for main- 
taining a mirrored image of storage on multiple volumes. 
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driver also supports the couplbg of a new store that is 
steadily broxi^t into synchrooicity with an existing store. 
Based on the mirroring algorithm being used by the driver 
and the status of the mirrored stores, the driver will route 
storage transactions to the appropriate virtual devices in the 5 
virtual circuit. Assuming that both the primary and alternate 
stores are functioning, the mirror driver will route this 
request to the primary and secondary stores only according 
to the parameter VD(15, 16, null, 17) or virtual devices 15 
and 16. The null in the parameter list indicates that no 10 
tertiary drive is currently being used for this virtual device. 

The mirroring driver may route the storage transaction 
messages in serial or in parallel to the two devices, la this 
example, the messaging to virtual device 15 will be consid- 
ered although (he example can also be extended to the 15 
second store, virtual device 16. Virtual device 15 includes 
redundant drivers for controlling a fibre channel drive. The 
drivers translate the internal format into a format used by the 
drives, e.g., BSA to SCSI. The drivers also provide the 
addressing information to the drive. Here, the parameter 20 
80(02) is used to select a storage option, here the fibre 
channel drive number 2. 

Accordingly, within the storage platform, hardware func- 
tions (such as disk or flash storage) and software functions 
(such as RAID stripes or Mirrors) arc all accessed via 
software drivers most commonly referred to as devices. 

These devices are paired up (with each member of the pair 
preferably running a separate board for redundancy) and 
called Mrtual Devices. Hiese Virtual Devices are then 
chained together into various configurations. For instance, a 
mirror device can be chained to two or three disk devices. 
Throu^ this type of configuration Virtual Device chains arc 
constructed. These Virtual Device chains can be added to as 
long as they arc being configiired into some BSA-typc 
device that can itself be used within yet another configura- 
tion. 

Virtual Device chains are connected to a FCP/SCSI Target 
Server device and mapped in the FCP Target Driver's LUN 
export tables for "export" (i.e., to be accessible via the FCP 
protocol from the outside worid). At that point the Virtual 
Device chain with a SCSI Target Server Device at its head 
is called a Virtual Circuit. 

The Virtual Circuit Manager software responsible for 
creating Virtual Circuits puts the SCSI Target Server "head" 
onto a Virtual Device chain and then exports the Virtual 
Circuit by updating the FCP Target's Export Tables. The 
software also supports delete, quiesce and fail-over opera- 
tions. 
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information, its status field is set to indicate the error 
and no further action is taken. 

2. Create a new SCSI Target Server device for the LUN 
of the Virtual Circuit specified by the newly inserted 
record. 

3. Set the status in the new record to "Instantiated". 

4. The storage assigned to the \^rtual Circuit will be 
flagged as used in a Storage Roll Call Table. 

5. The Export Table will be updated to dispatch the LUN 
to the new SCSI Target Server. 

When a record in the Virtual Circuit is deleted, the Virtual 
Circuit Manager will perform the following actions: 

1. Quiesce the Virtual Circuit if it isn't already and mark 
it as Quiesced. 

2. Remove the Virtual Circuit's dispatch data from the 
Export Table. 

3. Mark the Roll Call Record referenced from the Virtual 
Circuit Record as unused. 

4. Delnsiantiate the SCSI Target server associated with 
the Virtual Circuit. 

The Virmal Circuit Manager also listens for modifications 
to the "Exported" field in the VCT. If the "Exported" field 
in any record in the VCT is set to True, then the Virtual 
Circuit Manager will perform the following actions; 

1. Export the virtual circuit by making the necessary 
modifications to the FCP Target's Export Table. 

2. If there is any error encountered during the Export 
Operation, the status field in the VC Record will be set 
and the "Exported" field will be left in a correct state. 
If the Virtual Qrcuit was not exported, the Exported 
Flag will be set to False. 

The Virtual Circuit Manager listens for modifications to 
the "Quiesced" field in the Virtual Circuit Table. If the 
"Quiesced" field in any record in the VCT is set to True, then 
the Virtual Circuit Manager will perform the following 
actions: 

1. If the VC is currently exported, it will be unexported 
and its "Exported" flag will be set to False. 

2. All of the Virtual Devices in the Mrtual Circuit will be 
sent Quiesce Messages. 

3. If there is any error encountered during the Quiesce 
Operation, the status field in the VC record will be set 
and the "Quiesced" field will be left in a correct state, 
i.e., if the Virtual Circuit was not quiesced, the Qui- 
esced Flag will be set to False. 

FIG. 15 illustrates a storage area network utilizing a 
storage server 1200 according to the present invention. The 



maintaining the Virtual Circuit Tables VCTs that list in a 
single place all the Virtual Devices in each Virtual Circuit. 
This information is needed to implementing many system 
actions such as failovcr, hot-swap and shutdown. 



storage server 1200 in the network has client interfaces 
Virtual Circuity Manager software is alsojesponsible for ^210, 1211, 1212 coupled to client servers 1201, 1202, and 

1203 respectively. Storage interfaces 1213 and 1214 are 
coupled to communication channels to storage devices 1205, 
1206, 1207. The communication channel 1213 in this 
example is connected through a hub 1204 to the devices 
When it is initialized, the Virtual Circuit Manager soft- 55 i205 and 1206. In operation, the client interfaces operate 
ware defines the VCT itself in the persistent table store. Note according to protocol by which the client servers requests 
that this is harmless if the VCT has already been defined. storage transactions by commands which carry an identifier 
The Virtual Circuit Manager software also listens for of an initiator, a logical extent such as a LUN number, and 
insertions, deletions and any modifications to the VCT. an identifier of a target device. The storage server 1200 maps 

In order to create a new \^rtual Circuit, the information 60 in the requested transaction to a virtual device, which in turn 



necessary to instantiate a SCSI Target Server and to map and 
export the new LUN must be placed into a record in the 
VCT. The Virtual Circuit Manager listens for insertions into 
the VCT and upon receiving a listen reply will perform the 
following actions: 

1. Attempt to validate the information in the newly 
inserted record. If the record contains invalid 



allocates physical storage for use in the transaction from 
among the physical storage devices, The storage server 1200 
also includes resources that emulate the target physical 
device identified in the request. Tlie storage server 1200 is 
65 able to direct storage transactions using local configuration 
data, and simplify the management of storage for the client 
servers. 
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FIG. 16 illustrates another embodiment of a storage area 
network. In FIG. 18, a server 1250 which includes storage 
director logic and cache memory as discussed above, is 
coupled to client servers on a variety of different platforms, 
including a Hewlett-Packard server 1253, a Sun server 1256, 
and a SGI server 1257, each of which may be executing 
different protocols management of storage transactions. A 
plurality of physical storage devices is also coupled to the 
server 1250 and managed by the storage director according 



plurality of client servers 1310 through 1318 in this 
example. The client servers 1313 and 1314 are connected 
through a hub 1320 to the storage server 1301. Likewise, the 
client servers 1316 through 1318 arc connected to a hub 
1321 which in turn is connected lo the storage server 1302. 
The client servers 1310-1318 communicate with the storage 
server using storage channel protocols such as FCP 
described in detail above. According to these protocols, 
storage transactions are requested, and carry an identifier of 



to the virtual device architecture described above. The lo the initiator of the request, a logical unit number LUN, and 
plurality of physical storage devices in this example include an identifier of the target storage device. These parameters 
storage on a Hewlett-Packard platform 1251, storage on a are used by the storage director logic to map the storage 
Sun platform 1252 and a storage upon a EMC platform transaction to a virtual device. The servers also include 
1253. Thus, the server, including storage director logic, resources to emulate the target storage device so that the 
allows creation of a shared storage pool that can support IS client servers smoothly interoperate with the plurality of 
legacy servers and storage in a heterogeneous environment. storage devices in the storage area network. 
Incompatibilities among the plural storage devices and serv- In FIG. 17, there are a plurality of storage devices 1330 
ers can be masked or mimicked as needed using the virtual through 1339 illustrated coupled to the storage servers 
device architecture. True storage area network environments 1300-1302. In the diagram, a variety of symbols are used to 
can be implemented and all host, fabric and storage interop- 20 represent the storage devices, and to indicate that the net- 
erability issues can be managed at the storage server level. work is heterogeneous and can utilize a wide variety of 
The storage director logic utilizing the virtual device devices managed by the virtual device interfaces at the 
architecture provides a single intelligent coordination point servers 1301 through 1302. Also, the communication chan- 
for the configuration of server access to storage. Little or no nels can be varied. Thus, hubs 1340, 1341 and 1342 are 
hardware re-configuration is necessary in adding new 25 included in the network to facilitate a variety of communi- 
devices or changing management of existing devices. The 
configuration of the storage server provides accurate con- 
figuration information and control by allowing automatic 
maintenance of the mapping of data sets in physical storage 
to servers. Maintaining accurate map of physical storage 30 
simplifies management of storage area network significantly. 
Also, the storage director at the server provides for active 
migration of data fi-om old storage devices to new storage 
devices while the devices remain online. In addition, storage 
objects are no longer limited in size by the size of the largest 35 
object that can be created in an array. Multiple arrays can be 
concatenated into a single storage object independent of host 
operating systems running on client servers. The storage 
director can also manage backup and testing operations such 



as making snapshots of data in the non-volatile cache, and 40 gj^^jig^ ^jjg 



cation protocols between the storage devices and the storage 
servers. 

Conclusion 

An intelligent storage area network 0SAN) server archi- 
tecture has been described. The ISAN server architecture 
supports easy administration, uses standard components, 
provides an operating system, supports virtual devices, and 
provides high speed solid state drive systems. 

The foregoing description of various embodiments of the 
invention have been presented for purposes of illustration 
and description. The description is not intended to limit the 
invention to the precise forms disclosed. Many modifica- 
tions and equivalent arrangements will be apparent to people 



for managing data backup by copying data from disk to tape, 
for example, without being routed through the client server. 
Furthermore, the local cache can be used to migrate data 
from arrays that have lost redundancy and to repair redun- 
dant storage and maintain full availability of data while an 45 
array is being repaired or rebuilt. For applications having 
multiple servers accessing a common data set, locking logic 
can be placed in the storage server in a manner which 
provides a simple scalable solution using the virtual device 
architecture. 50 

The storage director logic in the storage server operates to 
consolidate caching requirements fi^om both' servers and 
storage to reduce the total amount of cache memory required 
for a storage area network. The system is able to allocate 
more cache to either the server the client server or storage 55 
system than either can effectively provide as internal 
memory. Further, the cache can be dynamically or statically 
allocated as defined for the applications using the system. 
FIG. 17 illustrates a more rigorous example of a storage 



What is claimed is: 

1. A storage server comprising: 

a plurality of communication interfaces; and 
data processing resources coupled vn\h the plurality of 
communication interfaces which transfer data among 
the plurality of communication interfaces, the data 
processing resources including a plurality of driver 
modules, and configurable logic linking driver modules 
in the plurality of driver modules into data paths, the 
data paths including respective sets of driver modules; 
wherein the configurable logic includes memory stor- 
ing lists of the respective sets of driver modules, and 
a user interface accepting data for the lists. 

2. The storage server of claim 1, wherein at least one data 
path comprises a plurality of driver modules each arranged 
lo provide redundant resources for a particular function. 

3. The storage server of claim 1, wherein at least one data 
path comprises a plurality of driver modules arranged to 



area network using a plurality of interconnected storage 60 provide redundant resources for functions in the at least one 



servers according to the present invention. Storage servers 
1300, 1301, and 1302 arc included, interconnected by com- 
munication channels 350, 351 using for example, a high 
speed protocol such as fiber channel. Gigabit cthernet, or 
ATM. Each storage server includes storage director logic 65 
and non-volatile cache in the preferred embodiment. The 
storage servers 1300, 1301, and 1302 are coupled to a 



data path. 

4. The storage server of claim 1, wherein the user inter- 
face comprises a graphical user interface. 

5. A storage server comprising: 

a plurality of communication interfaces; 
data processing resources coupled with the plurahty of 
communication interfaces which transfer data among 
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the plurality of commuoication interfaces, the data 
processing resources including a plurality of driver 
modules, and configurable logic linking driver modules 
in the plurality of driver modules into data paths, the 
data paths including respective sets of driver modules; 5 
and 

wherein the configurable logic includes memory stor- 
ing lists of the respective sets of driver modules, and 
a user interface accepting data for the lists; 

wherein the plurality of driver modules comprise lo 
instances of driver classes constructed in response to 
the configurable logic. 

6. A storage server comprising: 

a plurality of communication interfaces; 

data processing resources coupled with the plurality of 
communication interfaces which transfer data among 
the plurality of communication interfaces, the data 
processing resources including a plurality of driver 
modules, and configurable logic linking driver modules 
in the plurality of driver modules into data paths, the 
data paths including respective sets of driver modules; 
and 

wherein the configurable logic includes memory stor- 
ing lists of the respective sets of driver modules, and 
a user interface accepting data for the lists; 
the plurality of driver modules including one or more 
hardware driver modules to manage respective com- 
munication interfaces, and one or more internal driver 
modules to perform data path tasks independently of 
the plurality of communication interfaces. 

7. The storage server of claim 6, wherein the data path 
task of one or more internal driver modules includes cache 
memory management. 

8. The storage server of claim 6, wherein the data path 
task of one or more internal driver modules includes mirror 
storage management. 

9. The storage server of claim 6, wherein the data path 
task of one or more internal driver modules includes 
memory partition management. 

10. The storage server of claim 6, wherein the data path 
task of one or more internal driver modules includes data 
migration management. 

11. The storage server of claim 6, wherein the one or more 
internal driver modules includes a protocol server for a 
protocol supported on a communication interface in the 
plurality of communication interfaces. 

12. The storage server of claim 6, wherein the one or more 
hardware driver modules includes a physical layer driver for 
a communication link coupled to a communication interface 
in the plurality of communication interfaces. 

13. A storage server comprising: 

a plurality of communication interfaces; 

data processing resources coupled with the plurality of 
communication interfaces which transfer data among 55 
the plurality of communication interfaces, the data 
processing resources including a plurality of driver 
modules, and configurable logic linking driver modules 
in the plurality of driver modules into data paths, the 
data paths including respective sets of driver modules; 60 
and 

wherein the configurable logic includes memory stor- 
ing lists of the respective sets of driver modules, and 
a user interface accepting data for the lists; 

wherein the plurahty of driver modules includes a 65 
protocol server for a protocol supported on a com- 
munication interface in the plurality of communica- 
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tion interfaces, the protocol server recognizing target 
identifiers in a session according to the protocol and 
linking the session to a data path in response to the 
target identifier. 

14. The storage server of claim 13, wherein the protocol 
is compliant with a standard Internet Protocol (IP). 

15. The storage server of claim 13, wherein the commu- 
nication interface comprises an interface to a fibre chaimcl 
arbitrated loop. 

16. A storage server comprising: 

a plurality of communication interfaces; 

data processing resources coupled with the plurality of 
communication interfaces which transfer data among 
the plurality of communication interfaces, the data 
processing resources including a plurality of driver 
modules, and configurable logic linking driver modules 
in the plurality of driver modules into data paths, the 
data paths including respective sets of driver modules; 
and 

wherein the configurable logic includes memory stor- 
ing lists of the respective sets of driver modules, and 
a user interface accepting data for the lists; 

wherein driver modules in the plurality of driver mod- 
ules include logic for communication of data accord- 
ing to an internal message format, and include logic 
associated with the protocol server to translate data 
packets according to the protocol into an internal 
message format. 

17. The storage server of claim 16, wherein the internal 
message format comprises a message format supporting 
block transfers of 16 kilobytes or more. 

18. The storage server of claim 16, wherein the commu- 
nication interface comprises an interface to a fibre channel 
fabric. 

19. The storage server of claim 16, wherein the commu- 
nication interface comprises an interface to a point-to-point 
topology. 

20. A storage server comprising: 

a first plurality of communication interfaces coupled to 
respective host systems which execute data channel 
transactions; 

a second plurality of communication interfaces coupled to 

respective data storage devices; 
data processing resources coupled with the first and 
second pluralities of communication interfaces which 
transfer data among the plurality of communication 
interfaces, the data processing resources including 
a plurality of hardware driver modules to manage 
respective communication interfaces in the first and 
second plurality of communication interfaces, 
a plurality of internal driver modules to perform data 
path tasks, independent of the first plurality of com- 
munication interfaces, 
configurable logic linking driver modules in the plu- 
rality of hardware driver modules and the plurality of 
internal driver modules into data paths, the data 
paths including respective sets of one or more hard- 
ware driver modules and one or more internal driver 
modules, wherein the configurable logic includes a 
driver coupled to non-volatile memory to store tables 
identifying the respective sets of hardware driver 
modules and internal driver modules, and a user 
interface accepting data for the tables, and 
a protocol server module coupled with the data paths, for 
a protocol supported on a communication interface in 
the first plurality of communication interfaces, the 
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protocol server recognizing target identifiers in a ses- 26. The storage server of claim 21, wherein the data path 

sion according to the protocol and linking the session to task of one or more internal driver modules includes data 

a data path response to the target identifier. migration management. 

21. A storage server comprising: 27. The storage server of claim 21, wherein internal driver 
a plurality of communication interfaces; 5 modules in (he plurality of internal driver modules include 
data processing resources coupled with the plurahty of logic for communication of data according to an internal 

communication interfaces which transfer data among message format, and include logic associated with the pro- 

the plurality of communication interfaces, the data tocol server to translate data packets according to the 

processing resources including ^^^^^^^^ -^^^ ^ i^^^^,, (message format, 

a plurality of hardware driver modules to inanage ^S. The storage server of claim 21, wherein at least one 

respective communication mterraces m the plurality . , r j • j i t, 

c • • * p data path compnses a plurality oi driver modules each 

of coramunicadon interfaces, *^ . , , , • , 

a plurality of internal driver modules to perform data ^nrangcd to provide redundant resources for a particular 

path tasks, independently of the plurality of commu- function. 

nication interfaces, 29. The storage server of claim 21, wherein at least one 

configurable logic linking driver modules in the plu- data path comprises a plurality of driver modules arranged 

rality of hardware driver modules and the plurality of to provide redundant resources for functions in the at least 

internal driver modules into data paths, the data qhc data path. 

paths including respective sets of one or more hard- 3o_ xhe storage server of claim 21, wherein the protocol 

ware driver modules and one or more internal driver compliant with a standard Internet Protocol 0P). 

modules, and 31 -j^^ storage server of claim 21, wherein at least one of 

a protocol server module coupled with the data paths, for (he communication interfaces in the plurality of communi- 

a protocol supported on a communication interface in cation interfaces comprises an interface to a fibre channel 

the plurality of communication interfaces, the protocol ^ arbitrated loop. 

server recognizing target identifiere in a session accord- 32. xhe storage server of claim 27, wherein the internal 

ing to the protocol and Unking the session to a data path message format comprises a message format supporting 

in response to the target identifier. block transfers of 16 kilobytes or more. 

22. The storage server of clami 21, wherem the config- 33 ^ ^^^^^ ^^^^ ^2, ^^^^^.^ 

urable logic includes a driver coupled to non-volatile 3^ i^^^rface comprises a graphical user interfece. 

memory to store lists of the respective sets of driver J ^JJ^^ ^^^.^ ^^^^^^ 

modules, and a user interface accepting data for the lists. , , ■ j. , • • . *■ ■ * 1 

23. The storage server of claim 21, wherein the data path f . ^^"J^^ . ^^"^P^'^^ of mternal 
task of one or more intemal driver modules includes cache ^nver classes constmcted in response to the configurable 
memory management. ^^S^^- 

24. The storage server of claim 21, wherein the data path 35. The storage server of claim 21, wherein the protocol 
task of one or more intemal driver modules includes mirror server modules and the plurality of intemal driver modules 
storage management. comprise instances of internal driver classes constructed in 

25. The storage server of claim 21, wherein the data path response to the configurable logic, 
task of one or more intemal driver modules includes 

memory partition management. ♦ ♦ ♦ ♦ ♦ 
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